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Program Description 1 

Linked learning communities in postsecondary education 2 are pro- 
grams defined by having social and curricular linkages that provide 
undergraduate students with intentional integration of the themes and 
concepts that they are learning. 3 Linked learning communities are 
based on the theory that active learning in a community-based set- 
ting can improve academic outcomes by increasing social as well as 
academic integration. 4 To that end, linked learning communities tend 
to incorporate two characteristics: a shared intellectual theme with a 
linked or integrated curriculum and a community or common cohort 
of learners. 5 

Research 

The What Works Clearinghouse (WWC) identified six studies of linked 
learning communities in postsecondary education that both fall 
within the scope of the Developmental Students in Postsecondary 
Education topic area and meet WWC group design standards. All six 
studies meet WWC standards without reservations. Together, these 
studies included about 7,400 undergraduate students across six community colleges. 

The WWC considers the extent of evidence for linked learning communities to be medium to large for four outcome 
domains— academic achievement, postsecondary enrollment, credit accumulation, and progress in developmental 
education. These outcomes were assessed in all six of the studies that met WWC group design standards. The 
WWC considers the extent of evidence for linked learning communities to be small for one outcome domain — 
degree attainment. (See the Effectiveness Summary on p. 6 for further description of all five domains.) 

Effectiveness 

Overall, the effects of linked learning communities on academic achievement, degree attainment, postsecondary 
enrollment, credit accumulation, and progress in developmental education for postsecondary students were neither 
statistically significant nor large enough to be considered to be substantively important. Therefore, the WWC con- 
siders linked learning communities to have no discernible effects on these outcomes for community college stu- 
dents in developmental education. 
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Table 1. Summary of findings 6 




Improvement index (percentile points) 




Outcome domain 

Rating of 
effectiveness 

Average 

Range 

Number of 
studies 

Number of 
students 

Extent of 
evidence 

Academic achievement 

No discernible 
effects 

+3 

-1 to +4 

2 

2,241 

Medium to large 

Degree attainment 

No discernible 
effects 

+3 

na 

1 

1,534 

Small 

Postsecondary enrollment 

No discernible 
effects 

+1 

-1 to +4 

6 

7,419 

Medium to large 

Credit accumulation 

No discernible 
effects 

+1 

-1 to +3 

6 

7,419 

Medium to large 

Progress in developmental 
education 

No discernible 
effects 

+1 

-2 to +4 

6 

7,374 

Medium to large 


na = not applicable 
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Program Information 

Background 

There are many models of learning communities. Some of these are freshman seminars, team-taught courses, and 
residential-based learning communities. 7 These models draw in part from the residential college model utilized by 
Oxford University and Cambridge University in the United Kingdom. Learning communities were originally devel- 
oped by researchers and academics interested in reforming the instructional methods used at large US universities. 
The interest in reforming postsecondary education was driven in part by two major factors: a) the perceived need to 
improve learning outcomes for postsecondary students, and b) the desire to increase the probability that students 
starting college will actually finish. 8 

This intervention report focuses on a specific approach to learning communities called the linked learning commu- 
nities model. 9 In this model, cohorts of students enroll together in a foundational course (e.g., writing) and at least 
one additional content course (e.g., American history). The goal of this learning communities model is to improve 
student achievement through at least two mechanisms: a) shared curricular content (e.g., reading lists, assign- 
ments) across the courses, and b) co-registration with the same peer cohorts to provide social continuity within 
and between the linked courses. The idea is to give students the opportunity to apply the skills they learned in 
the foundational class to the content course’s assignments (for example, writing an expository essay on Nathaniel 
Hawthorne involves practicing skills learned in developmental writing). 

Program details 

The linked learning community models that were implemented in the six studies in this report were highly similar to 
one another in many important respects. They all took place in community colleges during similar time frames. All 
of the learning communities involved (at least, in theory) linkages across one developmental course and one or two 
other courses, and occurred within a single semester. Therefore, they all investigated the most basic form of the 
linked learning community model and also met the minimum requirements for inclusion in this review. 

The programs exhibited some variation across sites, including the specific nature of the sample, the focus of the 
developmental education experience (English or math), and the course to which the developmental education course 
was linked. In addition, the extent of curricular integration varied somewhat both across sites and over time. 10 


Cost 

Visher et al. (2012) report on implementation costs across three studies included in this intervention report (no cost 
information is available for the other studies). Implementation costs varied dramatically across sites, ranging from 
a low of $21 1 per student to a high of $1 ,190 per student (expressed in 201 1 dollars). The average cost across the 
three sites was $566 per student. About 43% of the implementation costs were attributed to additional instruction. 
Factors leading to additional instructional costs include compensation for the time needed for course planning and 
faculty collaboration, and smaller class sizes. Other cost categories were program support and student services 
(24%), which included case management and tutoring at some sites; administration (17%), which included a pro- 
gram coordinator and other administrative staff; and other services (16%), which varied across sites depending on 
the specific implementation parameters of the learning communities (e.g., one site provided students with vouchers 
for textbooks). 

Visher et al. (2012) also explore the cost effectiveness of the linked learning communities model as implemented in 
the studies included in this intervention report, a discussion of which is beyond the scope of a WWC review. Inter- 
ested readers are referred to Visher et al. (2012), pp. 67-76. 
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Research Summary 

The WWC identified 22 studies that investigated the effects of linked Table 2. Scope of reviewed research 
learning communities on the enrollment and achievement of postsec- 
ondary students in developmental education. 

The WWC reviewed all 22 of those studies against group design evi- 
dence standards. Six studies, reported in four different manuscripts 
(Sommo, Mayer, Rudd, & Cullinan, 2012; Weiss, Visher, & Wathington, 

2010; Weissman, Butcher, Schneider, Teres, Collado, Greenberg, & Welbeck, 2011; Weissman, Cullinan, Cerna, 
Safran, & Richman, 2012), are randomized controlled trials that meet WWC group design standards without reser- 
vations. These six studies are summarized in this report. Twelve studies do not meet WWC group design standards. 
Another four studies were determined not to meet WWC eligibility screens for review in this topic area. Citations for 
all 22 studies are in the References section, which begins on p. 10. 

Summary of studies meeting WWC group design standards without reservations 

Sommo et al. (2012) conducted a randomized controlled trial at Kingsborough Community College in Brooklyn, NY, 
with 1 ,534 students. During their first semester, 769 students were randomly assigned to participate in the Open- 
ing Doors Learning Communities program, which included an English course that was linked with two additional 
courses: an academic course required for the student’s major and a one-credit freshman orientation course. Addi- 
tionally, enhanced services (such as counseling/support services and textbook vouchers) were also available. The 
remaining 765 students were randomly assigned to the comparison group, who could not participate in the learning 
communities but did receive advice during the registration process from Opening Doors staff. Follow-up data were 
collected for up to 6 years after randomization. 

Weiss et al. (2010) conducted a randomized controlled trial at Hillsborough Community College in Tampa Bay, FL, 
with 1,071 students. The intervention group included 709 students who were randomly assigned to participate in 
the learning communities during their first semester at the college. The learning communities linked a College Suc- 
cess course with either College Preparatory Reading 1 or College Preparatory Reading 2, depending on how the 
student was placed. The comparison group included 362 students who did not participate in the learning commu- 
nities program. They were required to enroll in the College Success course, but they did not have to enroll during 
their first semesters. Follow-up data were collected for the first two semesters after the program. 

Weissman et al. (2011) conducted a randomized controlled trial with 1,273 students at Houston Community College 
in Houston, TX. The intervention group included 761 students randomly assigned to participate in the learning com- 
munities that were designed for students at the lowest level of developmental math offered by the college: Funda- 
mentals of Mathematics 1 (Math 1). Math 1 was linked with College and Career Planning, a student success course. 
The comparison group included 512 students who did not participate in the learning communities. Students in the 
comparison group were advised that they were required to take a developmental math course as a prerequisite to 
college-level math, but they were not mandated to do so. All first-year developmental students at the college were 
required to take the College and Career Planning course, including students in the comparison group, but for com- 
parison group students, this course was not intentionally linked to the developmental math course. College staff 
assisted students in both the intervention and comparison groups with registration for all their courses. Follow-up 
data were collected for one semester postprogram. 

Weissman et al. (2011) also conducted a randomized controlled trial at Queensborough Community College in 
Queens, NY, with 1 ,034 students. There were 608 students randomly assigned to the intervention group. The learn- 
ing communities at Queensborough linked two levels of developmental math (Basic Mathematics and Problem 
Solving and Elementary Algebra) to either developmental English or college-level English. The study was expanded 


Grade 

Postsecondary 

Delivery method 

Cohort 

Program type 

Practice 
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after the first semester so that the developmental math courses were linked to various other college-level courses 
(such as business and sociology). The comparison group consisted of 426 randomly assigned students who did 
not participate in the learning communities. Students in the comparison group were advised that they were required 
to take a developmental math course as a prerequisite to college-level math, but they were not mandated to do 
so. College staff assisted students in both the intervention and comparison groups with registration for all their 
courses. Follow-up data were collected for two semesters postprogram. 

Weissman et al. (2012) conducted a randomized controlled trial at the Community College of Baltimore County 
(CCBC) in Maryland with 1 ,083 students. The intervention group included 650 students who were randomly 
assigned to participate in the learning communities. The CCBC learning communities were organized around a 
developmental English course (either reading or writing). Students in the intervention group co-enrolled in the devel- 
opmental English course, a college-level content course, and a master learner session. The comparison group 
included 433 students who could not participate in the learning communities. Students in the comparison group 
enrolled in a credit-bearing student success course that was mandatory for all developmental reading students and 
was similar to the master learner session taken by students in the intervention group. The follow-up data were col- 
lected for the first semester after the program. 

Weissman et al. (2012) also conducted a randomized controlled trial at Merced College in California’s Central Val- 
ley with 1 ,424 students, 71 1 of whom were randomly assigned to the intervention group. Most of these students 
enrolled in a developmental writing course that was linked to (a) another developmental course in reading or math, 
(b) a college success course, or (c) an introductory college-level course (such as criminology or music), but one 
learning community linked developmental reading with a student success course. The comparison group included 
713 students who did not participate in a learning community. Follow-up data were collected for the first semester 
after the program. 

Summary of studies meeting WWC group design standards with reservations 

No studies of linked learning communities met WWC group design standards with reservations. 


Linked Learning Communities November 201 4 


Page 5 


WWC Intervention Report 


Effectiveness Summary 

The WWC review of linked learning communities for the Developmental Students in Postsecondary Education topic 
area includes student outcomes in five domains: academic achievement, degree attainment, postsecondary enroll- 
ment, credit accumulation, and progress in developmental education. The six studies of linked learning communi- 
ties that meet WWC group design standards reported findings in all five domains. The findings below present the 
authors’ estimates and WWC-calculated estimates of the size and statistical significance of the effects of linked 
learning communities on postsecondary students in developmental education. For a more detailed description of 
the rating of effectiveness and extent of evidence criteria, see the WWC Rating Criteria on p. 34. 


Summary of effectiveness for the academic achievement domain 

Two studies reported findings in the academic achievement domain. 

Sommo et al. (2012) reported the proportion of students with an overall grade point average of at least a C. Inter- 
vention group students were more likely to have an overall average of at least a C than were comparison group 
students (64% versus 59%). This effect was marginally statistically significant (p < .10), but it was not large enough 
to be considered substantively important. 

Weiss et al. (2010) reported no statistically significant difference in the proportion of students with an overall grade 
point average of at least a C either in the program semester (73% versus 72%) or in the first semester postprogram 
(65% versus 66%). Neither of these observed effect sizes was large enough to be considered substantively important. 

Thus, for the academic achievement domain, neither of the studies that met WWC group design standards showed 
either a statistically significant effect or an effect large enough to be considered substantively important. This 
results in a rating of no discernible effects, with a medium to large extent of evidence. 


Table 1. Rating of effectiveness and extent of evidence for the academic achievement domain 


Rating of effectiveness 

Criteria met 

No discernible effects 

None of the studies show 
a statistically significant or 
substantively important effect, 
either positive or negative. 

In the two studies that reported findings, the estimated impact of the intervention on outcomes in the academic 
achievement domain was neither statistically significant nor large enough to be substantively important. 

Extent of evidence 

Criteria met 

Medium to large 

Two studies that included about 2,300 students in two community colleges reported evidence of effectiveness in 
the academic achievement domain. 


Summary of effectiveness for the degree attainment domain 

One study reported findings in the degree attainment domain. 

Sommo et al. (2012) assessed the proportion of students who had attained a degree within 6 years after random 
assignment. Though a higher percentage of intervention group students had earned a degree (36% versus 31 %), 
this difference was only marginally statistically significant (p < .10), and it was not large enough to be considered 
substantively important. 

Thus, for the degree attainment domain, the one study that met WWC group design standards showed neither a 
statistically significant effect nor an effect large enough to be considered substantively important. This results in a 
rating of no discernible effects, with a small extent of evidence. 
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Table 2. Rating of effectiveness and extent of evidence for the degree attainment domain 


Rating of effectiveness 

Criteria met 

No discernible effects 

None of the studies show 
a statistically significant or 
substantively important effect, 
either positive or negative. 

In the one study that reported findings, the estimated impact of the intervention on outcomes in the degree 
attainment domain was neither statistically significant nor large enough to be substantively important. 

Extent of evidence 

Criteria met 

Small 

One study that included about 1 ,500 students in one community college reported evidence of effectiveness in the 
degree attainment domain. 


Summary of effectiveness for the postsecondary enrollment domain 

Six studies reported findings in the postsecondary enrollment domain. 

Sommo et al. (2012) found no statistically significant difference in the extent to which intervention and comparison 
group students enrolled in college (87% versus 85%) or in the extent to which the students registered for courses 
(93% versus 91 %). These differences are not large enough to be considered substantively important. 

Weiss et al. (2010) found no statistically significant differences in the proportion of students in the intervention 
and comparison groups registered for classes during the program semester (82% versus 83%), and this effect is 
not large enough to be considered substantively important. Women registered for courses at about the same rate 
irrespective of their condition (85% versus 84%), but men registered for courses at a lower rate in the intervention 
group (78%) than in the comparison group (83%). This difference in registration rates between men and women in 
the intervention group is not statistically significant, but is large enough to be considered substantively important. 
See Appendix D.1 for more details. 

Weissman et al. (2011) [Houston] found no statistically significant differences in the proportion of students in the 
intervention and comparison groups registered for classes during the program semester (84% versus 81 %); this 
effect is not large enough to be considered substantively important. These authors did observe a statistically signifi- 
cant interaction between student gender and learning communities on registration rates. Women in the intervention 
condition registered at a higher rate (88%) than women in the comparison condition (80%), while men in the inter- 
vention condition registered at a lower rate (78%) than men in the comparison condition (84%). This difference in 
registration rates between men and women in the intervention group is large enough to be considered substantively 
important. See Appendix D.1 for more details. 

Weissman et al. (2011) [Queensborough] found a marginally statistically significant difference in the proportion of 
students in the intervention and comparison groups registered for classes during the program semester (92% ver- 
sus 88%, p < .10). However, this difference is not large enough to be considered substantively important. 

Weissman et al. (2012) [Baltimore] found no statistically significant differences in the proportion of students in the 
intervention and comparison groups registered for classes during the program semester (85% versus 84%). This 
difference was not large enough to be considered substantively important. 

Weissman et al. (2012) [Merced] found no statistically significant differences in the proportion of students in the 
intervention and comparison groups registered for classes during the program semester (73% versus 74%). This 
difference was not large enough to be considered substantively important. 

Thus, for the postsecondary enrollment domain, none of the six studies that met WWC group design standards 
showed either a statistically significant effect or an effect large enough to be considered substantively important. 
This results in a rating of no discernible effects, with a medium to large extent of evidence. 
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Table 3. Rating of effectiveness and extent of evidence for the postsecondary enrollment domain 


Rating of effectiveness 

Criteria met 

No discernible effects 

None of the studies show 
a statistically significant or 
substantively important effect, 
either positive or negative. 

In the six studies that reported findings, the estimated impact of the intervention on outcomes in the postsecondary 
enrollment domain was neither statistically significant nor large enough to be substantively important. 

Extent of evidence 

Criteria met 

Medium to large 

Six studies that included about 7,400 students in six community colleges reported evidence of effectiveness in the 
postsecondary enrollment domain. 


Summary of effectiveness for the credit accumulation domain 

Six studies reported findings in the credit accumulation domain. 

Sommo et al. (2012) reported on the number of regular credits that students had accumulated within two semesters 
postprogram. Although the intervention group had accumulated more credits on average (27.7 credits versus 26.2 cred- 
its), this difference was not statistically significant, and was not large enough to be considered substantively important. 

Weiss et al. (2010) found no statistically significant difference in credit accumulation between intervention and com- 
parison students after the first semester postprogram (5.3 credits versus 4.9 credits), and the effect size is not large 
enough to be considered substantively important. 

Weissman et al. (2011) [Houston] found no difference in credit accumulation between intervention and comparison 
students after the second postprogram semester (4.7 credits in both groups). 

Weissman et al. (2011) [Queensborough] found no statistically significant difference in credit accumulation between 
intervention and comparison students after the second postprogram semester (12.4 credits versus 1 1 .8 credits), 
and the effect size is not large enough to be considered substantively important. 

Weissman et al. (2012) [Baltimore] found no statistically significant difference in credit accumulation between 
intervention and comparison students after the first semester postprogram (5.9 credits versus 6.0 credits), and the 
effect size is not large enough to be considered substantively important. 

Weissman et al. (2012) [Merced] found no statistically significant difference in credit accumulation between inter- 
vention and comparison students after the first semester postprogram (4.9 credits versus 5.1 credits), and the 
effect size is not large enough to be considered substantively important. 

Thus, for the credit accumulation domain, none of the six studies that met WWC group design standards showed 
either a statistically significant effect or an effect large enough to be considered substantively important. This 
results in a rating of no discernible effects, with a medium to large extent of evidence. 


Table 4. Rating of effectiveness and extent of evidence for the credit accumulation domain 


Rating of effectiveness 

Criteria met 

No discernible effects 

None of the studies show 
a statistically significant or 
substantively important effect, 
either positive or negative. 

In the six studies that reported findings, the estimated impact of the intervention on outcomes in the credit accumula- 
tion domain was neither statistically significant nor large enough to be substantively important. 

Extent of evidence 

Criteria met 

Medium to large 

Six studies that included about 7,400 students in six community colleges reported evidence of effectiveness in the 
credit accumulation domain. 
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Summary of effectiveness for the progress in developmental education domain 

Six studies reported findings in the progress in developmental education domain. 

Sommo et al. (2012) assessed whether students had passed both of their required developmental English assess- 
ment tests by the end of the second semester after the program. Although the intervention group did so at a higher 
rate (65% versus 60%), this effect was only marginally statistically significant (p < .10) and was not large enough to 
be considered substantively important. 

Weiss et al. (2010) reported no statistically significant difference in the proportion of students completing College 
Prep Reading (1 or 2) during the program semester (60% versus 59%); this effect is not large enough to be consid- 
ered substantively important. 

Weissman et al. (2011) [Houston] reported no statistically significant difference in the proportion of students pass- 
ing the second of two required developmental math courses within two semesters postprogram (18% versus 16%); 
this effect is not large enough to be considered substantively important. The study authors did observe a statisti- 
cally significant interaction between student gender and learning communities condition on progress through 
developmental education. Men passed the second of two required developmental math courses within two semes- 
ters at about the same rate regardless of condition (about 16% in both the intervention group and the comparison 
group). However, about 19% of women in the intervention group passed both courses, whereas 15% of women in 
the comparison group passed both courses. Even though this interaction is statistically significant, the difference in 
pass rates between men and women in the intervention group is not large enough to be considered substantively 
important. See Appendix D.2 for more details. 

Weissman et al. (2011) [Queensborough] reported no difference in the proportion of students in the intervention and 
comparison groups passing college math by the end of the second semester postprogram (10% in both groups). 

Weissman et al. (2012) [Baltimore] reported no statistically significant difference in the proportion of students in the 
intervention and comparison groups passing college composition by the end of the first semester postprogram 
(27% versus 30%). This effect size is not large enough to be considered substantively important. 

Weissman et al. (2012) [Merced] reported no statistically significant difference in the proportion of students in the 
intervention and comparison groups passing college composition by the end of the first semester postprogram 
(5.0% versus 5.1 %). This effect size is not large enough to be considered substantively important. 

Thus, for the progress in developmental education domain, none of the six studies that met WWC group design 
standards showed either a statistically significant effect or an effect large enough to be considered substantively 
important. This results in a rating of no discernible effects, with a medium to large extent of evidence. 


Table 5. Rating of effectiveness and extent of evidence for the progress in developmental education domain 


Rating of effectiveness 

Criteria met 

No discernible effects 

None of the studies show a statistically 
significant or substantively important 
effect, either positive or negative. 

In the six studies that reported findings, the estimated impact of the intervention on outcomes in the progress 
in developmental education domain was neither statistically significant nor large enough to be substantively 
important. 

Extent of evidence 

Criteria met 

Medium to large 

Six studies that included about 7,400 students in six community colleges reported evidence of effectiveness in 
the progress in developmental education domain. 
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college. Community College Review, 22(2), 16-25. The study does not meet WWC group design standards 
because it uses a quasi-experimental design in which the analytic intervention and comparison groups are not 
shown to be equivalent. 

VonHandorf, T. A. (201 2). Exploring the impact of learning communities at a community college: An effort to support 
students enrolled in a developmental math course (Doctoral dissertation). Available from ProQuest Disser- 
tations and Theses database. (UMI No. 3504070) The study does not meet WWC group design standards 
because it does not allow for an estimate of the intervention’s effect isolated from other factors. 

Studies that are ineligible for review using the Developmental Students in Postsecondary Education Evidence 
Review Protocol 

Ashley, W. J. (2012). The efficacy of learning communities in assisting developmental students in achieving gradu- 
ation and accumulation of credit hours in a southern metropolitan community college (Doctoral dissertation). 
Available from ProQuest Dissertation and Theses database. (UMI No. 3522188) The study is ineligible for 
review because it examines an intervention implemented in a way that does not fall within the scope of the 
review— the intervention is not a linked learning community as defined in this review (the learning community 
did not involve a specific linkage to another course). 

Green, M. L. (2006). A community college’s response to serving underprepared, nontraditional welfare-to-work stu- 
dents (Doctoral dissertation). Available from ProQuest Dissertation and Theses database. (UMI No. 3224155) 
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The study is ineligible for review because it does not examine an intervention implemented in a way that falls 
within the scope of the review— the intervention is not a linked learning community as defined in this review 
(the study was an evaluation of the New Visions program, which is a Welfare-to-Work program). 

Massie-Burrell, T. (201 0). A learning community project: Comparative interventions on writing apprehension and 
locus of control of developmental students at a two-year college (Doctoral dissertation). Available from Pro- 
Quest Dissertation and Theses database. (UMI No. 3443426) The study is ineligible for review because it does 
not include an outcome within a domain specified in the protocol. 

Visher, M. G., & Teres, J. (2011). Breaking new ground: An impact study of career-focused learning communities at 
Kingsborough Community College. New York: MDRC. http://files.eric.ed.gov/fulltext/ED522631.pdf. The study 
is ineligible for review because it does not use a sample aligned with the protocol— the students are not in 
developmental education. 
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Appendix A.1: Research details for Sommo et al. (2012) 

Sommo, C., Mayer, A. K., Rudd, T., & Cullinan, D. (2012). Commencement day: Six-year effects of a 
freshman learning community program at Kingsborough Community College. New York: MDRC. 
Retrieved from http://www.mdrc.org 


Table Al. Summary of findings Meets WWC group design standards without reservations 




Study findings 

Outcome domain 

Sample size 

Average improvement index 
(percentile points) 

Statistically significant 

Academic achievement 

1,409 students 

+4 

No 

Degree attainment 

1,534 students 

+3 

No 

Postsecondary enrollment 

1,534 students 

+2 

No 

Credit accumulation 

1,534 students 

+3 

No 

Progress in developmental 
education 

1,534 students 

+4 

No 


Setting The study took place at Kingsborough Community College, a large, urban community college 
in Brooklyn, NY, that is part of the City University of New York (CUNY) system. 

Study sample The Opening Doors Learning Communities program recruited students who met the following 
criteria: 1) first-time incoming freshmen who planned to attend college full time during the day; 
2) tested into developmental English (but did not test into English as a Second Language); 3) 
planned to attend college full time; and 4) between 1 7-34 years of age. 

The study initially enrolled students who were aged 18 or older but later enrolled students who 
were 17 years old with parental consent. Students initially had to report a household income 
below 250% of the federal poverty level, but this income criterion was also subsequently 
removed. Students in four career majors (accounting, business, mental health, and early child- 
hood education) were also excluded for the first year of the study because a separate learn- 
ing community operated for them. After the 2003-04 academic year, students in those career 
majors could participate in the Opening Doors program because the career learning commu- 
nity program ended. Students who were eligible were given the opportunity to participate in 
the study; 1,534 students were eligible to participate. Students were randomly assigned to the 
intervention and comparision conditions. After random assignment, 769 students were in the 
intervention group and 765 were in the comparison group. 

Among students in the sample, 55% were female, 38% were Black, 20% were Hispanic, and 
27% were White. Seventy-nine percent were between 17-20 years old, 91 % reported having 
no children, 28% indicated that their household was receiving government benefits (such as 
food stamps or Supplemental Security Income), 74% indicated they were financially depen- 
dent on their parents, 36% reported being currently employed, and 47% reported speaking a 
language other than English in their home. 
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Intervention 

group 


Comparison 

group 


Outcomes and 
measurement 


Support for 
implementation 


The Opening Doors Learning Communities program was organized around an English course, 
where the course level was determined by the students’ scores on the CUNY reading and writ- 
ing skills assessment tests administered before enrollment. The English course was linked with 
two additional courses: an academic course required for the student’s major and a one-credit 
freshman orientation course. The orientation course was available to all freshmen and teaches 
time management, study skills, college rules and procedures, and other topics relevant to new 
students. The three linked courses were taken together by groups of up to 25 students during 
their first semester in the study. The linked courses usually met one after the other. The Open- 
ing Doors Learning Communities operated only during a student’s first semester. 

Students in the learning communities were also offered other services, including 1) faculty col- 
laboration and instructional practices, 2) enhanced counseling and support services offered by 
a counselor/case manager, 3) enhanced tutoring for the English course (and, in some cases, 
the subject matter course), and 4) textbook vouchers for the initial program semester and sub- 
sequent winter or summer intersession. 

Over four semesters, the program included 40 learning communities: 31 with developmental 
English courses and 9 with college-level English courses. Learning community class sizes 
varied from 6-25 students, with an average of 17 students per learning community. 

Students assigned to the comparison group were enrolled in classes for which they were eli- 
gible or that were required, and they could receive the college’s standard services. In addition, 
similar to students in the intervention group, students in the comparison group were allowed 
to register for classes earlier than most freshmen, and they received advice on the registration 
process from Opening Doors staff. 

Researchers reported outcomes at nine points in time: the program semester (i.e., the semes- 
ter in which students were enrolled in a learning community), the first semester after the 
program, the second semester after the program, the third semester after the program, 2 
years after randomization, 3 years after randomization, 4 years after randomization, 5 years 
after randomization, and 6 years after randomization. Participation in the learning communities 
began in fall 2003, spring 2004, fall 2004, and spring 2005. For a more detailed description of 
these outcome measures, see Appendix B. 

According to the study authors, Kingsborough Community College provided 1 hour of reas- 
signed time for faculty to meet about course integration and support for students in learning 
communities (i.e., each 3-hour course was treated as a 4-hour course for purposes of determ- 
ing each faculty member’s teaching load). Each learning community also had an assigned 
tutor who attended the courses, and participating students received $150 textbook vouchers 
for the 12-week main session and a $75 textbook voucher for the subsequent 6-week winter 
or summer intersession for the campus bookstore. 
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Appendix A.2: Research details for Weiss et al. (2010) 

Weiss, M. J., Visher, M. G., & Wathington, M. (2010). Learning communities for students in developmen- 
tal reading: An impact study at Hillsborough Community College. New York: MDRC. http://files. 
eric.ed.gov/fulltext/ED510961.pdf. 


Table A2. Summary of findings Meets WWC group design standards without reservations 




Study findings 

Outcome domain 

Sample size 

Average improvement index 
(percentile points) 

Statistically significant 

Academic achievement 

832 students 

+1 

No 

Postsecondary enrollment 

1,071 students 

-1 

No 

Credit accumulation 

1,071 students 

+3 

No 

Progress in developmental education 

1,071 students 

+1 

No 


Setting The study took place at Hillsborough Community College (HCC), a large, urban community 

college in Tampa Bay, FL. The HCC has five campuses; three (Brandon, Dale Mabry, and Ybor 
City) participated in the Learning Communities Demonstration. 


Study sample To participate in the learning communities study at Hillsborough, students had to meet all of 

the following eligibility criteria: 1) at least 18 years old; 2) first-time students; and 3) placed into 
developmental reading— either College Preparatory Reading 1 or College Preparatory Read- 
ing 2 (i.e., one or two levels below college level). This level of placement was determined by 
scores on a state-mandated placement test. Students who were eligible were given the oppor- 
tunity to participate in the study (participation was voluntary). Randomization was done at the 
student level. At the start of the study, 1 ,071 students were eligible to participate in the study; 
709 were randomly assigned to the intervention group and 362 to the comparison group. 

Among students in the sample, 57% were female. Thirty-seven percent were Black, 32% were 
Hispanic, and 25% were White. Seventy percent of the students were between 17 and 20 
years old at the start of the study, 81 % reported having no children, 16% indicated that their 
household was receiving government benefits (such as food stamps or Supplemental Security 
Income), 25% indicated that they were receiving financial aid, 56% reported being currently 
employed, and 29% reported speaking a language other than English in their home. 

Intervention Students registered in a learning community that linked a College Success course with 

group either College Preparatory Reading 1 or College Preparatory Reading 2, depending on how 
the student was placed. Both reading courses emphasized vocabulary acquisition, reading 
comprehension, and writing to improve literacy development. The College Success course 
concentrated on a variety of academic and personal subjects, such as educational goals, 
planning, time management, study skills, health concerns, and career counseling. Students 
were also informed about other academic resources available to them. Students enrolled in 
the reading and College Success courses as a pair, creating the learning communities where 
the same small groups (cohorts) of students took the two linked courses together. Notably, 
however, the study authors reported that the linking of the curricula across the two courses 
did not occur until the study’s final semester. 
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Comparison Students assigned to the comparison group enrolled in any other classes for which they 

group were eligible or that were required, and they could receive the college’s standard services. All 

students who require one or more developmental courses must enroll in a College Success 
course; therefore, comparison group students had the option of enrolling in the course as well. 
However, they were not required to enroll immediately. 


Outcomes and Researchers reported outcomes at three points in time: the program semester (i.e., the semes- 
measurement ter during which students were enrolled in a learning community), the first semester after the 
program, and the second semester after the program. Participation in the learning communi- 
ties began in fall 2007, spring 2008, and fall 2008. For a more detailed description of these 
outcome measures, see Appendix B. 


Support for The learning community model was already in existence at Hillsborough when the study started, 
implementation According to the study authors, HCC supported a “paid coordinator who managed the program, 
strong support from college leadership, solid buy-in from faculty, and a collaborative relation- 
ship between academic and student affairs divisions.” The paid coordinator’s work from the first 
semester of implementation included organization of events and meetings for faculty develop- 
ment (e.g., strategies for active, collaborative learning and curricular integration), coordination of 
workshops and monthly meetings, and oversight of random assignment activities. 


Appendix A.3: Research details for Weissman et al. (2011) [Houston] 

Weissman, E., Butcher, K. F., Schneider, E., Teres, J., Collado, H., Greenberg, D., & Welbeck, R. (2011). 
Learning communities for students in developmental math: Impact studies at Queensborough 
and Houston Community Colleges. New York: National Center for Postsecondary Research, http:// 
files.eric.ed.gov/fulltext/ED516646.pdf. 

Table A3. Summary of findings Meets WWC group design standards without reservations 


Study findings 
Average improvement index 


Outcome domain 

Sample size 

(percentile points) 

Statistically significant 

Postsecondary enrollment 

1,273 students 

+2 

No 

Credit accumulation 

1,273 students 

+0 

No 

Progress in developmental 

1,273 students 

+2 

No 


education 


Setting The study took place at Houston Community College, a large community college system 

consisting of over 55,000 students attending six colleges located in and around Houston, TX. 
The learning communities project was conducted at three of the campuses around the city: 
Central, Northline, and Southeast. 
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Study sample 


Intervention 

group 


Comparison 

group 


Outcomes and 
measurement 


To be eligible for participation in the learning communities study, students had to meet all of 
the following criteria: 1) first-year student status; 2) placed into developmental math, including 
students in the lowest levels of developmental math (primarily Fundamentals of Mathematics 
I or Fundamentals of Mathematics II); and 3) available to take the learning community classes 
at their scheduled times. All students were placed at one of two developmental math levels 
based on a placement test (COMPASS). The study initially enrolled students who were 18 or 
older, but later enrolled students under 1 8. Across four semesters, 1 ,273 students were eligible 
to participate; 761 were randomly assigned the intervention group and 512 to the comparison 
group. Demographically, 67% of the entire sample of study participants were female, 55% 
were Hispanic, 35% were Black, and 3% were White. Sixty-three percent of the students in 
the sample were between 17 and 20 years old at the beginning of the study. In addition, 28% 
reported having at least one child, 29% were financially dependent on their parents, 41 % were 
currently employed, and 46% spoke a language other than English in their home. 

The learning communities at Houston Community College were designed for students at the 
lowest level of developmental math offered by the college: Fundamentals of Mathematics I 
(Math I). After passing Math I, students must pass both Fundamentals of Mathematics II and 
Intermediate Algebra before they can take a college-level course. Math I was linked with a 
student success course called College and Career Planning for the learning communities pro- 
gram. The student success course was designed to introduce students to tools and strategies 
that would help them achieve their college and career goals; included lessons on time man- 
agement, test-taking skills, and setting goals; and provided information on available campus 
resources such as tutoring. Math I is required for all students who place into it, but students 
are not required to take it during their first semester of enrollment. However, college policy 
does require all students to take the College and Career Planning course during their first 
semester. The key part of the learning community was the link between Math I and the student 
success course, and the linked courses included at least three integrated assignments. In 
addition, collaboration among faculty and connections to support services were reported by 
the study authors as salient aspects of the program. 

Comparison group students were advised that they were required to take developmental math 
courses as a prerequisite to college-level math, but they were not required to do so during the 
first semester. College staff assisted students in both conditions with registration for all of their 
courses. All first-year developmental students at the college were required to take the College 
and Career Planning course. 

The study authors gathered outcome data from transcripts provided by the college. The data 
were available at the end of the program semester and for one semester after the program. In 
addition, cumulative results at two semesters post-assignment were available. Participation 
in the learning communities began in spring 2008, fall 2008, spring 2009, and fall 2009. For a 
more detailed description of these outcome measures, see Appendix B. 
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Support for The study authors reported that each campus had a faculty member coordinator who received 

implementation course release time for their work on integrated curriculum planning. Faculty stipends varied 

from $200 to $400 per community taught. Campuses also varied in stipends for faculty plan- 
ning for new joint courses (e.g., $200) and one-course release time for planning the learning 
community the semester before. Additional supports to students included: math tutoring, ded- 
icated counseling, and development of online videos and PowerPoints into lessons accessed 
by students with purchased iPad touch devices. Field trips were supported by the campuses 
and incorporated into each class (one field trip per term). 


Appendix A.4: Research details for Weissman et al. (2011) [Queensborough] 

Weissman, E., Butcher, K. F., Schneider, E., Teres, J., Collado, H., Greenberg, D., & Welbeck, R. (2011). 
Learning communities for students in developmental math: Impact studies at Queensborough and 
Houston Community Colleges. New York: National Center for Postsecondary Research. http://files. 
eric.ed.gov/fulltext/ED516646.pdf. 

Table A4. Summary of findings Meets WWC group design standards without reservations 


Study findings 
Average improvement index 


Outcome domain 

Sample size 

(percentile points) 

Statistically significant 

Postsecondary enrollment 

1,034 students 

+4 

No 

Credit accumulation 

1,034 students 

+2 

No 

Progress in developmental education 

989 students 

-1 

No 


Setting The study took place at Queensborough Community College, a midsized community college 
located in Queens, NY, that is part of the City University of New York (CUNY) system. 

Study sample To be eligible for participation in the learning communities study, students had to meet all of 
the following criteria: 1) first-year student status, 11 2) placed into developmental math (includ- 
ing Basic Mathematics and Problem Solving or Elementary Algebra), and 3) available to take 
the learning community classes at their scheduled times. All students were placed at one of 
two developmental math levels based on a placement test (COMPASS). The study initially 
enrolled students who were 18 or older, but later enrolled students under 18. Across four 
semesters, 1 ,034 students were eligible to participate; 608 were randomly assigned to the 
intervention group and 426 to the comparison group. Demographically, 56% of the sample 
were female, 33% were Hispanic, 31 % were Black, 14% were White, and 12% were Asian/ 
Pacific Islander. Seventy-eight percent of the students in the sample were between 17 and 
20 years old at the beginning of the study. In addition, 7% reported having at least one child, 
37% were financially dependent on their parents, 40% were currently employed, and 39% 
spoke a language other than English in their home. 
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Intervention The learning communities were designed for students enrolled in the two levels of developmen- 

group tal math offered by the college: Basic Mathematics and Problem Solving (two levels below the 

first college-level class) and Elementary Algebra (one level below). During the first semester of 
the study (fall 2007), each of these math classes was linked with either developmental English 
or college-level English. To better match students’ needs and interest, during the remainder of 
the study (spring 2008, fall 2008, and spring 2009), the developmental math courses were linked 
with various college-level courses. In addition to the link with college-level English, other intro- 
ductory courses without prerequisites— such as business, sociology, and speech— were offered 
and linked with developmental math. This structure gave students the opportunity to move up in 
the developmental math sequence, while allowing them to earn college credits at the same time. 


Comparison Students assigned to the comparison group were advised that they were required to take 
group developmental math courses as a prerequisite to college-level math, but they were not 

required to do so during the first semester. College staff assisted students in both conditions 
with registration for all of their courses. 


Outcomes and The study authors gathered outcome data from transcripts provided by the college. The 

measurement outcomes focused on achievement in math and completion of both developmental and 

college-level courses in math and other subjects. The data were available for three semesters 
(the program semester and two semesters after the program). In addition, cumulative results 
at four semesters post-assignment were available. Participation in the learning communities 
began in fall 2007, spring 2008, fall 2008, and spring 2009. Outcomes were reported in three 
domains: postsecondary enrollment, progress in developmental education, and credit accu- 
mulation. For a more detailed description of these outcome measures, see Appendix B. 


Support for According to the study authors, beginning in the program’s second semester, leaders in academic 

implementation affairs and student services worked with the program coordinator, department chairs, and full-time 

faculty members to design and implement thematic learning community curricula. Faculty received 
professional development coaching that strengthened curricular integration, according to faculty 
surveys. Queensborough Community College provided faculty a stipend of $650 per semester. 


Appendix A.5: Research details for Weissman et al. (2012) [Baltimore] 

Weissman, E., Cullinan, D., Cerna, O., Safran, S., & Richman, P. (2012). Learning communities for stu- 
dents in developmental English: Impact studies at Merced College and the Community College of 
Baltimore County. New York: MDRC. http://files.eric.ed.gov/fulltext/ED529251.pdf. 


Table A5. Summary of findings Meets WWC group design standards without reservations 




Study findings 

Outcome domain 

Sample size 

Average improvement index 
(percentile points) 

Statistically significant 

Postsecondary enrollment 

1,083 students 

0 

No 

Credit accumulation 

1,083 students 

-1 

No 

Progress in developmental education 

1,083 students 

-2 

No 
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Setting 


Study sample 


Intervention 

group 


Comparison 

group 


Outcomes and 
measurement 


The study took place at the Community College of Baltimore County (CCBC), a community 
college located in Baltimore, MD, that has three campuses and three extension centers in sub- 
urban Maryland. Two campuses of CCBC participated in the Learning Communities Demon- 
stration: Catonsville and Essex. 

All new and returning students at CCBC were eligible to be included in the Learning Communi- 
ties Demonstration study sample if they had placed into a developmental English course (either 
reading or writing) that was one level below college-level English (resources were directed toward 
those students who had the highest chance of getting to college level). This level of placement 
was determined by Accuplacer test scores. Students also had to be available for class during the 
times that the learning community classes were offered. The study initially enrolled students who 
were 18 or older, but later enrolled students under 18 with parental consent. Students who were 
eligible were given the opportunity to participate in the study. Randomization was done at the stu- 
dent level. Across four semesters, 1 ,083 students were eligible to participate; 650 were randomly 
assigned to the intervention group and 433 to the comparison group. Demographically, 59% of the 
sample were female, 67% were non-White (predominately Black), and 77% were between 17 and 
20 years old. Furthermore, 16% reported having at least one child, 17% indicated that their house- 
hold was receiving government benefits (such as food stamps or Supplemental Security Income), 
46% indicated that they were receiving financial aid, 53% reported being currently employed, and 
8% reported speaking a language other than English in their home. 

The CCBC learning communities were organized around a developmental English course 
(either reading or writing). Students coenrolled in the developmental English course; a college- 
level content course (that was selected from a range of subject areas, such as sociology, 
psychology, or computer information); and a master learner session. The master learner ses- 
sion lasted for 1 hour per week and was a non-credit-bearing class that provided support and 
guidance as students worked through their linked courses. The session concentrated on help- 
ing students make connections between the content from the linked courses in each learning 
community and was designed to reinforce the instruction from those courses. 

Students assigned to the comparison group were allowed to enroll in any other classes for 
which they were eligible or that were required, and they could receive the college’s standard 
services. Many students in the comparison group enrolled in a credit-bearing student success 
course that was mandatory for all developmental reading students and that was, according to 
the study authors, similar in many respects to the master learner session taken by the learning 
community students in the intervention group. 

Researchers reported outcomes at two points in time: the program semester (i.e., the semester in 
which students were enrolled in a learning community) and the first semester after the program. At 
CCBC, participation in the learning communities began in spring 2008, fall 2008, spring 2009, and 
fall 2009. For a more detailed description of these outcome measures, see Appendix B. 
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Support for According to the study authors, support for the learning communities provided by CCBC 

implementation included: a cross-campus, lead program coordinator; support from a seasoned senior admin- 

istrator; and an experienced learning community coordinator at each campus. Program sup- 
port for faculty included professional development workshops on curricular integration and 
syllabi development. Faculty received a stipend of $750 or received a course load reduction 
equivalent to one credit hour for every learning community taught. Faculty who created new 
learning communities received an additional one-time stipend of $500 and $1 ,000, depending 
upon the degree of course integration. Faculty who taught master learner sessions received 
stipends of $2,250 or course load reductions of three credit hours. 


Appendix A.6: Research details for Weissman et al. (2012) [Merced] 

Weissman, E., Cullinan, D., Cerna, O., Safran, S., & Richman, P. (2012). Learning communities for stu- 
dents in developmental English: Impact studies at Merced College and the Community College of 
Baltimore County. New York: MDRC. http://files.eric.ed.gov/fulltext/ED529251.pdf 


Table A6. Summary of findings Meets WWC group design standards without reservations 




Study findings 

Outcome domain 

Sample size 

Average improvement index 
(percentile points) 

Statistically significant 

Postsecondary enrollment 

1,424 students 

0 

No 

Credit accumulation 

1,424 students 

-1 

No 

Progress in developmental education 

1,424 students 

0 

No 


Setting The study took place at Merced College, a midsized, rural community college located in Cali- 
fornia’s Central Valley enrolling over 10,000 students. 

Study sample All new and returning students were eligible to be included in the Learning Communities Demon- 
stration study sample if they had placed into developmental reading, writing, or both at one, two, 
or three developmental levels below Freshman Composition (the first college-level English course). 
Students assessed at four developmental levels below freshman composition (the lowest basic 
language and learning course) were not included in the study. This level of placement was deter- 
mined by Accuplacer test scores. Students also had to be available for class during the times that 
the learning community classes were offered. The study initially enrolled students who were 18 or 
older, but later enrolled students under 1 8 with parental consent. Students who were eligible were 
given the opportunity to participate in the study (participation was voluntary). Randomization was 
done at the student level. Across four semesters, 1 ,424 students were eligible to participate; 71 1 
were randomly assigned the intervention group and 713 to the comparison group. Demographi- 
cally, 49% of the sample were female, 83% were non-White (predominately Hispanic), and 65% 
were between 17 and 20 years old. Furthermore, 27% reported having at least one child, 37% 
indicated that their household was receiving government benefits (such as food stamps or Supple- 
mental Security Income), 28% indicated that they were receiving financial aid, 23% reported being 
currently employed, and 41 % reported speaking a language other than English in their home. 
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Intervention 

group 


Comparison 

group 

Outcomes and 
measurement 


Support for 
implementation 


The learning communities model at Merced College generally concentrated either on writing 
or on writing and reading combined. In the study, 1 1 of the 1 2 learning communities con- 
sisted of students who enrolled in a developmental writing course that was linked to either 
1) a developmental reading or math course, 2) a college success course, or (3) an introduc- 
tory college-level course (such as criminology, health, or music). One learning community 
linked developmental reading with a student success course. The learning communities were 
designed around a common theme shared between two courses (for example, writing about 
and understanding mathematical concepts). Students enrolled in two linked courses of a 
learning community formed a cohort, and they progressed together throughout the semester. 

Students assigned to the comparison group were allowed to enroll in any other classes for which 
they were eligible or that were required, and they could receive the college’s standard services. 

Researchers reported outcomes at two points in time: the program semester (i.e., the semester 
in which students were enrolled in a learning community) and the first semester after the pro- 
gram. Participation in the learning communities began in spring 2008, fall 2008, spring 2009, and 
fall 2009. For a more detailed description of these outcome measures, see Appendix B. 

According to the study authors, Merced College provided participating instructors with a 
$1 ,000 stipend for each learning community they taught per semester. However, over the 
course of the study, stipends for faculty who taught multiple learning communities each term 
were reduced. 
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Appendix B: Outcome measures for each domain 


Academic achievement 


Academic achievement 


Grade point average 

Taken from school administrative records, this outcome assesses the grades that students earned in credit- 
bearing (i.e., nondevelopmental) courses. 

Degree attainment 


Degree/certificate attainment 

Taken from administrative data and from the National Student Clearinghouse, this outcome assesses the extent 
to which students earned a formal degree or certificate. 

Postsecondary enrollment 


Registered for courses 

Taken from school administrative records, this outcome assesses whether students registered for courses. 

Enrolled in college 

Taken from National Student Clearinghouse records, this outcome assesses whether students actually enrolled 
in college. 

Credit accumulation 


College credits earned 

Taken from school administrative records, this outcome assesses the number of regular college credits earned 
by students. 

Progress in developmental education 


Passed all required developmental 
courses 

Taken from school administrative records, this outcome assesses whether students passed all required develop- 
mental courses (e.g,, both levels of developmental math for students needing two levels of remediation). 

Passed targeted college-level 
introductory course 

Taken from school administrative records, this outcome assesses whether students passed the targeted college- 
level introductory course (i.e., college-level English or college-level math). 

Passed developmental education tests 

Taken from school administrative records, this outcome assesses whether students were able to pass develop- 
mental skills assessment tests. 


Degree attainment 


Postsecondary enrollment 


Credit accumulation 


Progress in developmental education 
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Appendix C.1: Findings included in the rating for the academic achievement domain 




Mean 

(standard deviation) 

WWC calculations 


Outcome measure 

Study 

sample 

Sample Intervention 
size group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Sommo et al. (2012) a 

Proportion of students earning at 
least a C average 

College 

students 

1,409 64% 

59% 

4% 

0.11 

4 

nr 

Domain average for academic achievement (Sommo et al., 2012) 




+4 

nr 

Weiss etal. (201 0) b 

Proportion of students earning at 
least a C average (program semester) 

College 

students 

832 73% 

72% 

1% 

0.04 

2 

> .10 

Proportion of students earning at 
least a C average (first semester 
postprogram) 

College 

students 

591 65% 

66% 

-1% 

-0.04 

-1 

> .10 

Domain average for academic achievement (Weiss et al., 2010) 




+1 

>.10 

Domain average for academic achievement across all studies 




+3 

na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on individual outcomes, representing the average change expected for all individu- 
als who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting 
the change in an average individual’s percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average 
rounded to two decimal places; the average improvement index is calculated from the average effect size. The statistical significance of each study’s domain average was deter- 
mined by the WWC. Some statistics may not sum as expected due to rounding, na = not applicable, nr = not reported. 

a For Sommo et al. (201 2), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-values presented here were 
reported in the original study. This outcome is calculated by adjusting the percentages reported in Scrivener et al. (2008; Table 4.4) by removing those students with no grade point 
average (GPA). This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor large enough to be substantively 
important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 

b For Weiss et al. (201 0), a correction for multiple comparisons was needed but did not affect whether any of the contrasts were found to be statistically significant. This outcome is calcu- 
lated by removing students with no GPA from the sample size, and combining the reported proportions of students with GPAs higher than a C average adjusted for the appropriate sample 
size. The p-values presented here were reported in the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically 
significant nor large enough to be substantively important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 


Appendix C.2: Findings included in the rating for the degree attainment domain 





Mean 

(standard deviation) 

WWC calculations 


Study 

Sample 

Intervention 

Comparison 

Mean Effect Improvement 

Outcome measure 

sample 

size 

group 

group 

difference size index p-value 


Sommo etal. (2012) a 

Earned a degree (within 6 College 1,534 36% 

years of randomization) students 

31% 

5% 

0.09 

3 

<.10 

Domain average for degree attainment (Sommo et al., 2012) 




3 

<.10 

Domain average for degree attainment across all studies 




+3 

na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on individual outcomes, representing the average change expected for all individu- 
als who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the 
change in an average individual’s percentile rank that can be expected if the individual is given the intervention, na = not applicable. 

a For Sommo et al. (201 2), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was reported 
in the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor large enough to be substan- 
tively important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 
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Appendix C.3: Findings included in the rating for the postsecondary enrollment domain 


Mean 

(standard deviation) 

WWC calculations 


Study Sample Intervention Comparison 

Outcome measure sample size group group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Sommo et al. (2012) a 

Enrolled in college College 1,534 87% 85% 

students 

3% 

0.07 

3 

>.10 

Registered for courses College 1,534 93% 91% 

students 

2% 

0.06 

2 

>.10 

Domain average for postsecondary enrollment (Sommo et al., 2012) 



2 

>.10 

Weiss etal. (201 0) b 

Registered for courses College 1,071 82% 83% 

students 

-1% 

-0.03 

-1 

>.10 

Domain average for postsecondary enrollment (Weiss et al., 2010) 



-1 

>.10 

Weissman et al. (2011) [Houston] 6 

Registered for courses College 1,273 84% 81% 

students 

3% 

0.06 

2 

>.10 

Domain average for postsecondary enrollment (Weissman et al., 2011) [Houston] 



2 

>.10 

Weissman et al. (2011) [0ueensborough] d 

Registered for courses College 1,034 92% 88% 

students 

4% 

0.11 

4 

<.10 

Domain average for postsecondary enrollment (Weissman et al., 2011) [Queensborough] 



4 

<.10 

Weissman et al. (2012) [Baltimore] 6 

Registered for courses College 1,083 85% 84% 

students 

0% 

0.00 

0 

>.10 

Domain average for postsecondary enrollment (Weissman et al., 2012) [Baltimore] 



0 

>.10 

Weissman et al. (2012) [Merced]' 

Registered for courses College 1,424 73% 74% 

students 

-1% 

-0.01 

0 

>.10 

Domain average for postsecondary enrollment (Weissman et al., 2012) [Merced] 



0 

>.10 

Domain average for postsecondary enrollment across all studies 



+1 

na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on individual outcomes, representing the average change expected for all individu- 
als who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the 
change in an average individual's percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average round- 
ed to two decimal places; the average improvement index is calculated from the average effect size. The statistical significance of each study’s domain average was determined by 
the WWC. Some statistics may not sum as expected due to rounding, na = not applicable. 

a For Sommo et al. (2012), the Enrolled in college outcome was reported in the supplemental study by Scrivener et al. (2008; Table 4.2), and the Registered for courses outcomes was 
reported in the supplemental study by Bloom and Scrivener (2005; Table 5). A correction for multiple comparisons was needed but did not affect whether any of the contrasts were 
found to be statistically significant. The p-values presented here were reported in the original studies. This study is characterized as having indeterminate effects because the reported 
effect sizes are neither statistically significant nor large enough to be substantively important. For more information, please refer to the WWC Standards and Procedures Handbook, 
version 3.0, p. 26. 

b For Weiss et al. (201 0), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was reported in 
the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor large enough to be substantively 
important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 
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c For Weissman et al. (201 1 ) [Houston], no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here 
was reported in the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor large enough to 
be substantively important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 

11 For Weissman et al. (2011) [Queensborough], no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value pre- 
sented here was reported in the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor 
large enough to be substantively important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 

e For Weissman et al. (201 2) [Baltimore], no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here 
was reported in the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor large enough to 
be substantively important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 

f For Weissman et al. (201 2) [Merced], no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here 
was reported in the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor large enough to 
be substantively important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 


Appendix C.4: Findings included in the rating for the credit accumulation domain 


Mean 

(standard deviation) 


WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Sommo etal. (201 2) a 

Regular credits earned 

College 

students 

1,534 

27.7 

(nr) 

26.2 

(nr) 

1.5 

0.07 

3 

>.10 

Domain average for credit accumulation (Sommo et al., 2012) 




3 

>.10 

Weiss etal. (201 0) b 

Regular credits earned 

College 

students 

1,071 

5.3 

(nr) 

4.9 

(nr) 

0.4 

0.08 

3 

>.10 

Domain average for credit accumulation (Weiss et al., 2010) 




3 

>.10 


Weissman et al. (2011) [Houston] 0 


College 

students 


1,273 


Regular credits earned 

Domain average for credit accumulation (Weissman et al., 2011) [Houston] 


4.7 

(nr) 


4.7 

(nr) 


0.00 


>.10 

>.10 


Weissman et al. (2011) [Queensborough] d 


College 

students 


1,034 


Regular credits earned 

Domain average for credit accumulation (Weissman et al., 2011) [Queensborough] 


12.4 

(nr) 


11.8 

(nr) 


0.6 


0.04 


>.10 

>.10 


Weissman et al. (2012) [Baltimore] 6 


Regular credits earned College 1,424 

students 

Domain average for credit accumulation (Weissman et al., 2012) [Baltimore] 


5.9 

(nr) 


6.0 

(nr) 


- 0.1 


- 0.02 


-1 

-1 


>.10 

>.10 


Weissman et al. (2012) [Merced]' 

Regular credits earned 


College 

students 


1,083 


4.9 

(nr) 


5.1 

(nr) 


- 0.2 


- 0.02 


Domain average for credit accumulation (Weissman et al., 2012) [Merced] 


Domain average for credit accumulation across all studies 


-1 

-1 


>.10 

>.10 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on individual outcomes, representing the average change expected for all individu- 
als who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting 
the change in an average individual's percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average 
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rounded to two decimal places; the average improvement index is calculated from the average effect size. The statistical significance of each study’s domain average was deter- 
mined by the WWC. Some statistics may not sum as expected due to rounding, na = not applicable, nr = not reported. 

a For Sommo et al. (2012), regular credits earned were reported in the supplemental study by Scrivener et al. (2008; Table 4.4). No corrections for clustering or multiple comparisons 
and no difference-in-differences adjustments were needed. The p-value presented here was reported in the original study. This study is characterized as having indeterminate effects 
because the reported effect sizes are neither statistically significant nor large enough to be substantively important. For more information, please refer to the WWC Standards and 
Procedures Handbook, version 3.0, p. 26. 

b For Weiss et al. (201 0), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was reported in 
the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor large enough to be substantively 
important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 

c ForWeissman et al. (2011) [Houston], no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here 
was reported in the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor large enough to 
be substantively important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 

d ForWeissman et al. (2011) [Queensborough], no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value pre- 
sented here was reported in the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor 
large enough to be substantively important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 

e For Weissman et al. (201 2) [Baltimore], no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here 
was reported in the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor large enough to 
be substantively important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 

f For Weissman et al. (201 2) [Merced], no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here 
was reported in the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor large enough to 
be substantively important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 


Appendix C.5: Findings included in the rating for the progress in developmental education domain 


Mean 

(standard deviation) 

WWC calculations 


Study Sample Intervention Comparison 

Outcome measure sample size group group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Sommo et al. (2012) a 

Passed both developmental College 1,534 65% 60% 

education tests students 

5% 

0.10 

4 

<.10 

Domain average for progress in developmental education (Sommo et al., 2012) 



4 

<.10 

Weiss etal. (201 0) b 

Completed college College 1,071 60% 59% 

preparatory reading students 

2% 

0.03 

1 

>.10 

Domain average for progress in developmental education (Weiss et al., 2010) 



1 

>.10 

Weissman et al. (2011) [Houston] 0 

Passed both required College 1,273 18% 16% 

developmental math courses students 

2% 

0.06 

2 

>.10 

Domain average for progress in developmental education (Weissman et al., 2011) [Houston] 


2 

>.10 

Weissman et al. (2011) [Queensborough] d 

Passed college math College 989 10% 10% 

students 

0% 

-0.01 

-1 

>.10 

Domain average for progress in developmental education (Weissman et al., 2011) [Queensborough] 


-1 

>.10 

Weissman et al. (2012) [Baltimore] 6 

Passed college English College 1,083 27% 30%o 

students 

-3% 

-0.06 

-2 

>.10 

Domain average for progress in developmental education (Weissman et al., 2012) [Baltimore] 


-2 

>.10 
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Weissman et al. (2012) [Merced]' 

Passed college English College 1,424 5% 5% 0% 

students 

0 

0 

>.10 

Domain average for progress in developmental education (Weissman et al., 2012) [Merced] 


0 

>.10 

Domain average for progress in developmental education across all studies 


+1 

na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on individual outcomes, representing the average change expected for all individu- 
als who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the 
change in an average individual's percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average round- 
ed to two decimal places; the average improvement index is calculated from the average effect size. The statistical significance of each study’s domain average was determined by 
the WWC. Some statistics may not sum as expected due to rounding, na = not applicable. 

a For Sommo et al. (2012), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was reported 
in the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor large enough to be substan- 
tively important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 

b For Weiss et al. (2010), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was reported in 
the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor large enough to be substantively 
important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 

c For Weissman et al. (201 1 ) [Houston], no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here 
was reported in the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor large enough to 
be substantively important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 

d For Weissman et al. (201 1 ) [Queensborough], no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value pre- 
sented here was reported in the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor 
large enough to be substantively important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 

e For Weissman et al. (2012) [Baltimore], no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here 
was reported in the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor large enough to 
be substantively important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 

f For Weissman et al. (201 2) [Merced], no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here 
was reported in the original study. This study is characterized as having indeterminate effects because the reported effect sizes are neither statistically significant nor large enough to 
be substantively important. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, p. 26. 
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Appendix D.1: Summary of subgroup findings for the academic achievement domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Sommo et al. (2012) a 

Proportion of students 
earning at least a C average 

Women 

775 

66% 

64% 

2% 

0.06 

2 

nr 

Proportion of students 
earning at least a C average 

Men 

634 

60% 

54% 

6% 

0.16 

6 

nr 

Weiss etal. (201 0) b 

Proportion of students 
earning at least a C average 
(program semester) 

Women 

488 

77% 

76% 

1% 

0.04 

1 

nr 

Proportion of students 
earning at least a C average 
(program semester) 

Men 

344 

68% 

65% 

3% 

0.07 

3 

nr 

Proportion of students 
earning at least a C average 
(first semester postprogram) 

Men 

237 

60% 

65% 

-5% 

-0.13 

-5 

nr 


Table Notes: The supplemental findings presented in this table are additional subgroup findings from the studies in this report that do not factor in the determination of the 
intervention rating. The subgroup analyses examine the effects of the intervention by gender. Total group scores were used for rating purposes and are presented in Appendices 
C.1-C.5. For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors the 
comparison group. The effect size is a standardized measure of the effect of an intervention on individual outcomes, representing the average change expected for all individu- 
als who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting 
the change in an average individual's percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average 
rounded to two decimal places; the average improvement index is calculated from the average effect size, nr = not reported. 

a For Sommo et al. (201 2), the gender subgroup analyses were reported in the supplemental Scrivener et al. (2008) study in Appendix Table D.4. No corrections for clustering or 
multiple comparisons and no difference-in-differences adjustments were needed. This calculation removed students with no GPA from the sample, and combined the proportions of 
students reporting at least a C average while adjusting for the analysis sample size. The study did not report p-values for this outcome as presented here. 

b For Weiss et al. (201 0), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The gender subgroup data included were 
provided by the authors following an author query. The outcomes reported remove students with no GPA and calculate the proportion of students earning at least a C average by 
combining different ranges reported and adjusting for the analysis sample size. The proportion of students earning at least a C average in the first semester postprogram for women 
was not included because it did not meet WWC group design standards, owing to attrition. No p-values were reported in the additional data. 
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Appendix D.2: Summary of subgroup findings for the postsecondary enrollment domain 




Mean 

(standard deviation) 

WWC calculations 



Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect Improvement 
size index 

p-value 

Sommo et al. (2012) a 

Enrolled In college 

Women 

837 

87% 

84% 

3% 

0.08 

3 

>.10 

Enrolled In college 

Men 

697 

88% 

85% 

2% 

0.07 

3 

>.10 

Registered for courses 

Women 

837 

94% 

92% 

2% 

0.06 

2 

>.10 

Registered for courses 

Men 

697 

92% 

90% 

2% 

0.06 

2 

>.10 

Weiss etal. (201 0) b 

Registered for courses 

Women 

609 

85% 

84% 

1% 

0.02 

1 

nr 

Registered for courses 

Men 

462 

78% 

83% 

-5% 

-0.13 

-5 

nr 


Weissman et al. (2011) [Houston] c 


Registered for courses 

Women 

849 

88% 

80% 

8% 

0.17 

7 

< .05 

Registered for courses 

Men 

424 

78% 

84% 

-6% 

-0.13 

-5 

>.10 


Weissman et al. (2011) [Queensborough] d 


Registered for courses 

Women 

579 

92% 

88% 

4% 

0.12 

5 

>.10 

Registered for courses 

Men 

455 

92% 

88% 

4% 

0.12 

5 

>.10 


Weissman et al. (2012) [Baltimore] 6 


Registered for courses 

Women 

636 

85% 

85% 

0% 

0.00 

0 

>.10 

Registered for courses 

Men 

447 

84% 

84% 

0% 

-0.01 

0 

>.10 


Weissman et al. (2012) [Merced] e 


Registered for courses 

Women 

730 

75% 

75% 

0% 

-0.01 

0 

>.10 

Registered for courses 

Men 

694 

72% 

72% 

-1% 

-0.01 

0 

>. 10 


Table Notes: The supplemental findings presented in this table are additional subgroup findings from the studies in this report that do not factor in the determination of the inter- 
vention rating. The subgroup analyses examine the effects of the intervention by gender. Total group scores were used for rating purposes and are presented in Appendices C.1- 
C.5. For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors the com- 
parison group. The effect size is a standardized measure of the effect of an intervention on individual outcomes, representing the average change expected for all individuals who 
are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change 
in an average individual’s percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average rounded to 
two decimal places; the average improvement index is calculated from the average effect size. Some statistics may not sum as expected due to rounding, nr = not reported. 
a For Sommo et al. (2012), the gender subgroup analyses were reported in the supplemental Scrivener et al. (2008) study. No corrections for clustering or multiple comparisons and no 
difference-in-differences adjustments were needed. The p-values presented here were reported in the original study. The college enrollment outcomes were reported in Appendix D.6 
of the study, and the course registration outcomes were reported in Table D.2 of the Scrivener et al. (2008) report. 

b For Weiss et al. (2010), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The data included were provided by the 
authors following an author query. No p-values were reported in the additional data, but WWC-calculated p-values were not significant at the .05 level. 

c For Weissman et al. (201 1 ) [Houston], no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The gender subgroup out- 
comes were reported in Table A.1 0 of the study. The p-values presented here were reported in the original study. 

d For Weissman et al. (201 1 ) [Queensborough], no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The gender subgroup 
outcomes were reported in Table A.5 of the study. The p-values presented here were reported in the original study. 

e For Weissman et al. (2012), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The gender subgroup data included 
were provided by the authors following an author query. The p-values presented here were reported by the authors. 
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Appendix D.3: Summary of subgroup findings for the credit accumulation domain 


Outcome measure 

Study 

sample 

Sample 

size 

Mean 

(standard deviation) 

Intervention Comparison 
group group 

WWC calculations 

Mean Effect Improvement 

difference size index 

p-value 

Sommo etal. (201 2) a 

Regular credits earned 

Women 

837 

28.9 

28.5 

0.4 

0.01 

1 

>.10 




(nr) 

(nr) 





Regular credits earned 

Men 

697 

26.4 

23.5 

2.9 

0.14 

5 

<.10 




(nr) 

(nr) 





Weiss etal. (201 0) b 

Regular credits earned 

Women 

609 

5.6 

5.4 

0.2 

0.04 

2 

nr 




(5.5) 

(5.9) 





Regular credits earned 

Men 

462 

5.0 

4.2 

0.8 

0.15 

6 

nr 




(5.4) 

(5.2) 





Weissman et al. (2011) [Houston] 0 

Regular credits earned 

Women 

849 

5.2 

5.0 

0.2 

0.05 

2 

>.10 




(nr) 

(nr) 





Regular credits earned 

Men 

424 

3.5 

4.3 

-0.8 

-0.16 

-6 

>.10 




(nr) 

(nr) 





Weissman et al. (2011) [Queensborough] d 

Regular credits earned 

Women 

579 

13.2 

11.8 

1.4 

0.11 

4 

>.10 




(nr) 

(nr) 





Regular credits earned 

Men 

455 

11.4 

11.7 

-0.3 

-0.02 

-1 

>.10 




(nr) 

(nr) 






Table Notes: The supplemental findings presented in this table are additional subgroup findings from the studies in this report that do not factor in the determination of the 
intervention rating. The subgroup analyses examine the effects of the intervention by gender. Total group scores were used for rating purposes and are presented in Appendices 
C.1-C.5. For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors the 
comparison group. The effect size is a standardized measure of the effect of an intervention on individual outcomes, representing the average change expected for all individu- 
als who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting 
the change in an average individual's percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average 
rounded to two decimal places; the average improvement index is calculated from the average effect size, nr = not reported. 

a For Sommo et al. (2012), the gender subgroup analyses were reported in the supplemental Scrivener et al. (2008) study. No corrections for clustering or multiple comparisons and no 
difference-in-differences adjustments were needed. The p-values presented here were reported in the original study. 

b For Weiss et al. (201 0), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The gender subgroup data included were 
provided by the authors following an author query. No p-values were reported in the additional data, but WWC calculated p-values were not significant at the .05 level. 


c For Weissman et al. (201 1 ), the subgroup outcomes for the Houston site were reported in Table A.1 0 of the study. No corrections for clustering or multiple comparisons and no 
difference-in-differences adjustments were needed. The p-values presented here were reported in the original study. 

d For Weissman et al. (2011), the subgroup outcomes for the Queensborough site were reported in Table A.5 of the study. No corrections for clustering or multiple comparisons and no 
difference-in-differences adjustments were needed. The p-values presented here were reported in the original study. 
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Appendix D.4: Summary of subgroup findings for the progress in developmental education domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Sommo etal. (201 2) a 

Passed both developmental 
education tests 

Women 

837 

69% 

67% 

2% 

0.04 

2 

>.10 

Passed both developmental 
education tests 

Men 

697 

60% 

52% 

8% 

0.16 

6 

<.05 

Weiss etal. (201 0) b 

Completed college 
preparatory reading 

Women 

609 

71% 

68% 

4% 

0.08 

3 

nr 

Completed college 
preparatory reading 

Men 

462 

62% 

57% 

5% 

0.10 

4 

nr 


Weissman et al. (2011) [Houston] 0 


Passed both required 
developmental math courses 

Women 

849 

19% 

15% 

4% 

0.10 

4 

>.10 

Passed both required 
developmental math courses 

Men 

424 

16% 

16% 

-1% 

-0.01 

-1 

>.10 


Weissman et al. (2011) [Queensborough] d 


Passed college math 

Women 

561 

12% 

9% 

3% 

0.11 

5 

>.10 

Passed college math 

Men 

428 

8% 

13% 

-5% 

-0.16 

-6 

<.10 


Weissman et al. (2012) [Baltimore] 6 


Passed college English 

Women 

636 

29% 

33% 

-4% 

-0.08 

-3 

>.10 

Passed college English 

Men 

447 

24% 

25% 

-1% 

-0.02 

-1 

>.10 


Weissman et al. (2012) [Merced] 6 


Passed college English 

Women 

730 

7% 

6% 

0% 

0.01 

0 

>.10 

Passed college English 

Men 

694 

4% 

4% 

0% 

0.00 

0 

>.10 


Table Notes: The supplemental findings presented in this table are additional subgroup findings from the studies in this report that do not factor in the determination of the inter- 
vention rating. The subgroup analyses examine the effects of the intervention by gender. Total group scores were used for rating purposes and are presented in Appendices C.1- 
C.5. For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors the com- 
parison group. The effect size is a standardized measure of the effect of an intervention on individual outcomes, representing the average change expected for all individuals who 
are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change 
in an average individual's percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average rounded to 
two decimal places; the average improvement index is calculated from the average effect size. Some statistics may not sum as expected due to rounding, nr = not reported. 
a For Sommo et al. (201 2), the gender subgroup analyses were reported in the supplemental Scrivener et al. (2008) study in Appendix Table D.5. No corrections for clustering or mul- 
tiple comparisons and no difference-in-differences adjustments were needed. The p-values presented here were reported in the original study. 

b For Weiss et al. (201 0), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The data included were provided by the 
authors following an author query. No p-values were reported in the additional data, but WWC calculated p-values were not significant at the .05 level. Data in this table reflect the 
cumulative rates for completion of developmental reading. 

c For Weissman et al. (201 1 ), the subgroup outcomes for the Houston site were reported in Table A.9 of the study. No corrections for clustering or multiple comparisons and no 
difference-in-differences adjustments were needed. The p-values presented here were reported in the original study. 

d For Weissman et al. (2011), the subgroup outcomes for the Queensborough site were reported in Table A.4 of the study. No corrections for clustering or multiple comparisons and no 
difference-in-differences adjustments were needed. The p-values presented here were reported in the original study. 

e For Weissman et al. (201 2), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The gender subgroup data and the 
p-values presented here were provided by the authors. 
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Endnotes 

1 Linked learning communities do not have a single developer or official description. The descriptive information for this program was 
obtained from publicly available sources, including the research articles reviewed in this report (e.g., Inkelas & Soldner, 2011; Lenning 
& Ebbers, 1999). The literature search was completed in March 2013 and updated in August 2013. 

2 Postsecondary education refers to any formal education beyond high school; most commonly, college or university. 

3 Inkelas, K. K., & Soldner, M. (201 1). Undergraduate living-learning programs and student outcomes. In J. Smart & M. Paulsen (Eds.), 
Handbook of theory and research (pp. 335-368). New York: Springer. 

4 Smith, B. L., MacGregor, J., Matthews, R. S., & Gabelnick, F. (2004). Learning communities: Reforming undergraduate education. 

San Francisco, CA: Wiley. 

5 E.g., Lenning, O. T., & Ebbers, L. H. (1999). The powerful potential of learning communities: Improving education for the future. 
ASHE-ERIC Higher Education Report, 26(6). Washington, DC: Graduate School of Education and Human Development, George 
Washington University, http://eric.ed. gov/?id=ED428606; Shapiro, N. S., & Levine, J. H. (1999). Creating learning communities: A 
practical guide to winning support, organizing for change, and implementing programs. San Francisco, CA: Jossey-Bass. http://eric. 
ed.gov/?id=ED434624. 

6 For criteria used in the determination of the rating of effectiveness and extent of evidence, see the WWC Rating Criteria on p. 34. These 
improvement index numbers show the average and range of student-level improvement indices for all findings across the studies. 

7 For typologies, see Gabelnick, F., MacGregor, J., Matthews, R. S., & Smith, B. L. (1990). Learning communities: Creating connections 
among students, faculty, and disciplines. New Directions for Teaching and Learning, 4 7 (spring). San Francisco, CA: Jossey-Bass; 
Inkelas, K. K., & Soldner, M. (201 1). Undergraduate living-learning programs and student outcomes. In J. Smart & M. Paulsen (Eds.), 
Handbook of theory and research (pp. 335-368). New York: Springer; Lenning, O. T., & Ebbers, L. H. (1999). The powerful potential 

of learning communities: Improving education for the future. ASHE-ERIC Higher Education Report, 26(6). Washington, DC: Gradu- 
ate School of Education and Human Development, George Washington University, http://eric.ed. gov/?id=ED428606; Shapiro, N. S., 

& Levine, J. H. (1 999). Creating learning communities: A practical guide to winning support, organizing for change, and implementing 
programs. San Francisco, CA: Jossey-Bass. http://eric.ed. gov/?id=ED434624; Smith, B. L., MacGregor, J., Matthews, R. S., & Gabel- 
nick, F. (2004). Learning communities: Reforming undergraduate education. San Francisco, CA: Wiley. 

8 Inkelas, K. K., & Soldner, M. (201 1). Undergraduate living-learning programs and student outcomes. In J. Smart & M. Paulsen (Eds.), 
Handbook of theory and research (pp. 335-368). New York: Springer. 

9 A term adapted from Smith, B. L., MacGregor, J., Matthews, R. S., & Gabelnick, F. (2004). Learning communities: Reforming under- 
graduate education. San Francisco, CA: Wiley. 

10 Readers interested in more information regarding the challenges associated with implementing the basic linked learning community 
model and potential solutions to those challenges are referred to Visher, M. G., Schneider, E., Wathington, H., & Collado, H. (2010). 
Scaling up learning communities: The experience of six community colleges. New York: MDRC. http://eric.ed. gov/?id=ED509307. 

11 Returning students who had failed developmental math and transfer students who had fewer than 15 credits were also eligible to 
participate at Queensborough. 
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U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2014, November). 
Developmental Students in Postsecondary Education intervention report: Linked learning communities. 
Retrieved from http://whatworks.ed.gov 


Linked Learning Communities November 201 4 


Page 33 


WWC Intervention Report 


WWC Rating Criteria 

Criteria used to determine the rating of a study 

Study rating 

Criteria 

Meets WWC group design 
standards without reservations 

A study that provides strong evidence for an intervention’s effectiveness, such as a well-implemented RCT. 

Meets WWC group design 

A study that provides weaker evidence for an intervention's effectiveness, such as a QED or an RCT with high 

standards with reservations 

attrition that has established equivalence of the analytic samples. 

Criteria used to determine the rating of effectiveness for an intervention 

Rating of effectiveness 

Criteria 

Positive effects 

Two or more studies show statistically significant positive effects, at least one of which met WWC group design 
standards for a strong design, AND 

No studies show statistically significant or substantively important negative effects. 

Potentially positive effects 

At least one study shows a statistically significant or substantively important positive effect, AND 

No studies show a statistically significant or substantively important negative effect AND fewer or the same number 

of studies show indeterminate effects than show statistically significant or substantively important positive effects. 

Mixed effects 

At least one study shows a statistically significant or substantively important positive effect AND at least one study 
shows a statistically significant or substantively important negative effect, but no more such studies than the number 
showing a statistically significant or substantively important positive effect, OR 

At least one study shows a statistically significant or substantively important effect AND more studies show an 
indeterminate effect than show a statistically significant or substantively important effect. 

Potentially negative effects 

One study shows a statistically significant or substantively important negative effect and no studies show 
a statistically significant or substantively important positive effect, OR 

Two or more studies show statistically significant or substantively important negative effects, at least one study 
shows a statistically significant or substantively important positive effect, and more studies show statistically 
significant or substantively important negative effects than show statistically significant or substantively important 
positive effects. 

Negative effects 

Two or more studies show statistically significant negative effects, at least one of which met WWC group design 
standards for a strong design, AND 

No studies show statistically significant or substantively important positive effects. 

No discernible effects 

None of the studies show a statistically significant or substantively important effect, either positive or negative. 

Criteria used to determine the extent of evidence for an intervention 

Extent of evidence 

Criteria 

Medium to large 

The domain includes more than one study, AND 
The domain includes more than one school, AND 

The domain findings are based on a total sample size of at least 350 students, OR, assuming 25 students in a class, 
a total of at least 14 classrooms across studies. 

Small 

The domain includes only one study, OR 
The domain includes only one school, OR 

The domain findings are based on a total sample size of fewer than 350 students, AND, assuming 25 students 
in a class, a total of fewer than 14 classrooms across studies. 
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Glossary of Terms 

Attrition 

Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Extent of evidence 

Improvement index 

Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Rating of effectiveness 

Single-case design 
Standard deviation 


Statistical significance 


Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

An indication of how much evidence supports the findings. The criteria for the extent 
of evidence levels are given in the WWC Rating Criteria on p. 34. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

The WWC rates the effects of an intervention in each domain based on the quality of the 
research design and the magnitude, statistical significance, and consistency in findings. The 
criteria for the ratings of effectiveness are given in the WWC Rating Criteria on p. 34. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < .05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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